Dynamic membership management in a distributed system

ABSTRACT

Transactional database replication techniques are disclosed that do not require altering of the database management system implementation. A replicator module includes a dynamic membership manager, which is configured to manage the persistent membership of a coordination group. The persistent membership can be, for example, a durable set of sites that can replicate changes amongst themselves. Changes from sites not in the persistent membership are discarded. The set of recovered members is a subset of the persistent membership. The persistent membership changes incrementally by either removing or adding members. Failed members may still be part of the persistent membership. The dynamic membership manager module manages the modification of the persistent membership, initialization of replicas, and propagation of membership information.

RELATED APPLICATIONS

This application is a continuation of prior U.S. application Ser. No.11/404,155, filed Apr. 13, 2006, which claimed the benefit of U.S.Provisional Application No. 60/671,373, filed on Apr. 13, 2005, which isherein incorporated in its entirety by reference. In addition, thisapplication is related to U.S. application Ser. No. 11/270,196, filedNov. 8, 2005, and titled “Fault Tolerant Distributed Lock Management”and U.S. application Ser. No. 11/292,055, both filed Nov. 30, 2005, andtitled “Update-Anywhere Replication of Distributed Systems.” Each ofthese applications is herein incorporated in its entirety by reference.

FIELD OF THE INVENTION

The invention relates to distributed systems, and more particularly, totransactional database replication.

BACKGROUND OF THE INVENTION

Modern computer systems typically consist of a CPU to process data, anetworking interface to communicate to other computer systems, and oneor more durable storage units. The system may stop processing due topower failure, program incorrectness, or a hardware fault. Such failuresare often called process failures. The durable storage units are able tokeep the data intact while the fault is repaired.

A set of these computer systems can be networked to form a cluster.Although the network is generally reliable, occasional faults may occurto disrupt communication between certain nodes or sets of nodes. Thisdisruption in communication is often called a network partition.

Each of these nodes runs a transactional storage system that both readsand writes data (a database management system). Some of this data isconcurrently accessed by applications operating on different nodes. Toguarantee data consistency, transactional database replicationtechniques are used to manage and regulate access to that data. However,such conventional replication techniques are associated with a number ofproblems.

For instance, conventional replication systems typically require anadministrator to stop all applications, then stop the databases runningat each of the nodes in the replicating group, then go to each node andnotify it about the change in membership (usually by editing a table),then restarting the databases on each of these machines, then restartingthe applications. This necessity of having to stop the databases runningin the system can be undesirable.

What is needed, therefore, are transactional database replicationtechniques that do not require stopping of databases during replication.

SUMMARY OF THE INVENTION

One embodiment of the present invention provides a method fordynamically managing persistent membership of a coordination group in adistributed environment. The coordination group includes a number ofsites included in a persistent membership, with one or more of the sitesincluding a database and a registrar. The method includes sending (by anentrant seeking to become a member of the persisted membership) aMembership Request, and receiving (by a registrar that is a member ofthe persistent membership) the Membership Request. The method continueswith modifying (by the registrar) the persistent membership for theentrant. Once the persistent membership has changed, the methodcontinues with disconnecting (by the registrar) from the coordinationgroup, and initializing a database for the entrant. In one particularembodiment, modifying the persistent membership for the entrant includessending (by the registrar) a Membership Change message to other sites inthe coordination group, and then receiving (by the registrar) theMembership Change message in total persisted order. The method continueswith proceeding (by the registrar and/or entrant) with initializing theentrant database in response to the persistent membership being changed.The method may include reconnecting (by the registrar) to thecoordination group after initialization of the entrant database, andconnecting (by the entrant) to the coordination group afterinitialization of the entrant database. In one particular case, shouldconnection between the registrar and the entrant be severed, the entrantcan reconnect to any available registrar via a group communicationsystem, and recover. In another particular case, initializing theentrant database is carried out using backup/historical initialization.In such a case, the Membership Request includes an action ID of a lastaction that was applied to the entrant's database, the action IDuniquely identifying the persistent membership. In response to theregistrar not containing historical update information for this actionID, the method may further include refusing (by the registrar) toproceed. The method may include acquiring (by the registrar) adistributed lock on the coordination group and entrant name, and thenexamining (by the registrar upon receiving the distributed lock) thepersistent membership to ensure that no other site utilizes theentrant's name, and then refusing (by the registrar) to proceed withinitializing the database in response to the entrant's name beingalready in use. In one such case, acquiring the distributed lock iscarried out by a fault tolerant distributed lock manager associated withthe registrar. Here, the method may further include aborting (by thefault tolerant distributed lock manager) any transactions in process,and releasing (by the fault tolerant distributed lock manager) any locksheld by the registrar, and disconnecting (by the registrar) from a groupcommunication system providing an extended virtual synchrony service tothe coordination group thereby allowing the registrar to provide aconsistent snapshot to the entrant. In another particular case,initializing the entrant database is carried out using snapshotinitialization, and the entrant's database is initialized with contentsof the registrar's database. In another particular case, initializingthe entrant database is carried out using verify/update initialization.Here, the initializing may further include comparing entrant andregistrar database contents, and resolving discrepancies. In response tothe entrant including new data, the method may further include creating(by the entrant) actions corresponding to the new data and indicatingtotal persisted order of those actions is not yet known. In anotherparticular case, initializing the entrant database is carried out usingbackup/historical initialization. Here, the initializing may furtherinclude disconnecting (by the registrar) from a group communicationsystem servicing the coordination group thereby inhibiting furtherupdate actions, and sending (by the registrar) all Green actions afterlast applied action at the entrant to the entrant (where Green actionsare those whose total persisted order is known and confirmed), andapplying (by the entrant) those sent Green actions. The method mayfurther include inhibiting (by a distributed lock manager) transactionsduring database initialization. In another particular case, theregistrar can only service an entrant if the registrar has attainedquorum based on a quorum requirement, and the quorum requirement can beloosened to allow more quorums by not counting previous quorum membersthat have left or have been evicted. In another particular case, membersare allowed to become registrars only if they have participated in atleast one quorum. The method may include using a flow control protocolso as not to exceed any network data flow limits.

Another embodiment of the present invention provides a machine-readablemedium (e.g., one or more compact disks, diskettes, servers, memorysticks, or hard drives) encoded with instructions, that when executed byone or more processors, cause the processor to carry out a process fordynamically managing persistent membership of a coordination group in adistributed environment (recall that a coordination group includes anumber of sites included in a persistent membership, with one or more ofthe sites including a database and a registrar). This process can be,for example, similar to or a variation of the previously describedmethod.

Another embodiment of the present invention provides a system fordynamically managing persistent membership of a coordination group in adistributed environment. The system functionality (e.g., such as that ofthe previously described method or a variation thereof) can beimplemented with a number of means, such as software (e.g., executableinstructions encoded on one or more computer-readable mediums), hardware(e.g., gate level logic or one or more ASICs), firmware (e.g., one ormore microcontrollers with I/O capability and embedded routines forcarrying out the functionality described herein), or some combinationthereof

The features and advantages described herein are not all-inclusive and,in particular, many additional features and advantages will be apparentto one of ordinary skill in the art in view of the figures anddescription. Moreover, it should be noted that the language used in thespecification has been principally selected for readability andinstructional purposes, and not to limit the scope of the inventivesubject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system for replicating autonomousdatabases distributed across a network, configured in accordance with anembodiment of the present invention.

FIG. 2 a is a block diagram of a replicator module shown in FIG. 1,configured in accordance with an embodiment of the present invention.

FIG. 2 b is a block diagram of a dynamic membership manager shown inFIG. 2 a, further illustrating data flow between the dynamic membershipmanager and other components of the replicator module shown in FIG. 2 a,in accordance with an embodiment of the present invention.

FIG. 3 illustrates a registrar state machine of the dynamic membershipmanager shown in FIGS. 2 a and 2 b, configured in accordance with anembodiment of the present invention.

FIG. 4 illustrates an entrant database initialization state machine ofthe dynamic membership manager shown in FIGS. 2 a and 2 b, configured inaccordance with an embodiment of the present invention.

FIG. 5 illustrates a registrar database initialization state machine ofthe dynamic membership manager shown in FIGS. 2 a and 2 b, configured inaccordance with an embodiment of the present invention.

FIG. 6 illustrates an entrant state machine of the dynamic membershipmanager shown in FIGS. 2 a and 2 b, configured in accordance with anembodiment of the present invention.

FIG. 7 illustrates a membership leave state machine of the dynamicmembership manager shown in FIGS. 2 a and 2 b, configured in accordancewith an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Transactional database replication techniques are disclosed that employa dynamic membership management. Dynamic membership management allowsmodification of membership of a coordination group without interruptingreplication availability for the general membership.

General Overview

Each of the “sites” in a network of computer systems can run atransactional storage system (also called a database management system).In one embodiment of the present invention, the data in each of thesestorage systems is replicated to some or all of the others. There can beseveral applications concurrently updating and reading the data at aparticular site. Also, the sites can operate autonomously and beconcurrently updated with respect to each other. Updates made at onesite can be transferred to the other sites. To each application orapplication instance, the replica of the database appears as if it werethe only database in the system. Effective transactional databasereplication is provided.

One such embodiment integrates a fault tolerant lock manager (such asthe one described in the previously incorporated U.S. application Ser.No. 11/270,196) with a persistent total order algorithm to build atransactional replication system that does not require the modificationof the databases. In one particular case, the manner of this integrationinvolves an organization of functions including database transactioncommit, transaction commitment notification to the replication system,capture and recording of changes made to the data stored in thedatabase, retrieving the changes from the database, sending the changesto the other replicas of the system, and transferring associated locksto others as needed. The set of sites containing the various replicas istermed a “coordination group.”

In addition, a system resulting from this integration is a generalreplication system capable of operating with “off the shelf” databasemanagement systems, with performance and reliability characteristicssuperior to earlier designs. This embodiment of the invention issuitable, for example, for applications such as highly availableclusters, update-anywhere data caching, as well as heterogeneousdatabase replication. The database management systems employed in asingle coordination group may utilize the relational model, or otherdata models such as those provided by object databases, simultaneously.

The replication techniques employed can be method-based (such thosedescribed in the previously incorporated U.S. application Ser. No.11/292,055). There are many ways to realize these replication techniquesin an implemented system. In one particular implementation, the majorityof the functional logic is encapsulated in a module referred to hereinas a replicator. The replicator module includes a dynamic membershipmanager (DMM), which is configured to manage the persistent membershipof a coordination group. In one particular configuration, the persistentmembership is a durable set of servers (or other sites) that canreplicate changes amongst themselves. Changes from servers not in thepersistent membership are discarded. The set of recovered members is asubset of the persistent membership. The persistent membership changesincrementally by either removing or adding members. Failed members maystill be part of the persistent membership. In short, the DMM modulemanages the modification of the persistent membership, initialization ofreplicas, and propagation of membership information.

For ease of exposition, the replicator module is specified in detailbased on its inputs, outputs, and logic implemented within thereplicator. Numerous implementations and configurations will be apparentin light of this disclosure, and the present invention is not intendedto be limited to any one particular such embodiment. The replicationtechniques described herein can be used, for instance, to replicatedatabase management systems distributed across and connected by amessage passing network, and support a variety of data storagetechnologies such as traditional durable transactional databases as wellas in-memory data storage systems, and a variety of data types such asrelational, flat file, XML, and object-oriented data. The techniquesdescribed herein also provide database applications guaranteed levels ofdistributed transactional data consistency (e.g., serializable) for bothreads and writes. The techniques also support the ability ofapplications distributed across a network to perform updatesconcurrently, and the ability to replicate many standard databasesystems without needing to modify their intrinsic behavior or needingthem to be aware of the replicator, and the ability to tolerate networkand process failures without any change to the data consistencyguarantees provided to the applications. In short, dynamic membershipmanagement is enabled, which can be used in conjunction with faulttolerant, update-anywhere, transactional replication of autonomousdatabases distributed across and connected by a message-passing network,thereby providing a robust distributed system. Numerous computingenvironments and applications that can benefit from the techniquesdescribed herein will be apparent in light of this disclosure.

System Architecture

FIG. 1 is a block diagram of a system for replicating autonomousdatabases distributed across a network, configured in accordance withone embodiment of the present invention.

As can be seen, the system includes a number (N) of sites (or node) eachcommunicatively coupled to a network via an extended virtual synchrony(EVS) transport module. Each site includes one or more applications thatcan access the local database of that site (e.g., database A of site 1,database B of site 2, and database C of site N). In addition,applications from one particular site can access the databases of othersites. Thus, the resources stored in any one database can beconcurrently accessed by applications operating on different sites. Theapplications at each site can be executed, for example, by a processingsystem (not shown) such as a server, work station, laptop, personaldigital assistant, or any kind of computing system.

A replicator module local to each database is programmed or otherwiseconfigured to replicate the set of databases with each other. Eachinstance of the replicator module executes identical logic, as will bedescribed herein. Note, however, that correct operation of the logicrequires certain configuration information unique to each instance. Inparticular, each replicator module instance is configured to beassociated with its own database instance as well as its own uniqueidentifier. A particular instance of the replicator module configured inthis manner is termed a member (e.g., replicator members A, B, and C).

Recall that a set of sites including the database replicas is referredto herein as a coordination group. Any member belongs to a coordinationgroup. The replicator module replicates the data in the databases of allmembers that are in the same coordination group. Not all sites need areplicator module, unless there is only one site in the coordinationgroup. However, it is also acceptable for each site to have its ownreplicator module.

In addition to the associated unique identifier and database, eachinstance of the replicator module is configured to locally store apersistent membership list including the unique identifiers of all ofthe members of the coordination group. The contents of the membershiplists associated with each member belonging to the same coordinationgroup are identical. This membership list is managed by a dynamicmembership module as will be described in turn.

Each database may have multiple applications that concurrently query andupdate the data stored in the database. Note that in this exampleconfiguration, an application notifies the replicator module via areplicator API of transaction begins, commits, and lock requests.However, this direct communication from the application to thereplicator module is not necessary to the design of the replicatormodule. For instance, in an alternative embodiment, the database can beconfigured or modified to directly inform the replicator module of thesenotifications. In another embodiment, the database driver or provider(e.g., JDBC, .Net, ODBC, or other middleware) can be programmed orotherwise configured to intercept operations (e.g., begin, update,commit) from an application. In such a configuration, thedriver/provider can be further configured to inform the replicatormodule of necessary notifications. Another embodiment is to leveragestandard database customization facilities like stored procedures andtriggers to provide the replicator module with necessary notifications.For example, update triggers can be used to provide the replicatormodule with an OnLock event.

If a coordination group contains more than one member, the memberscommunicate to each other via the communications network (e.g., TCP/IP,NFS, Ethernet, etc), as shown in FIG. 1. The network may intermittentlyfail, partitioning the members into a variety of configurations. Forexample, the initial configuration {A,B,C} may be partitioned into twoconfigurations: {A,B} and {C}. In this use of the replicator module, thethree applications may concurrently query and update their respectivedatabases, as shown in FIG. 1. The replicator module distributes thesechanges and updates of data from the applications such that there is nosemantic difference from the applications point of view in terms of dataconsistency in this system as compared to a system with a singledatabase that is receiving concurrent queries and updates from multipleapplications.

This equivalence in data consistency from the perspective of theapplications is what is meant by transactional replication. Althoughthere is equivalence in data consistency, there are significantadvantages in terms of performance and availability in the embodimentdepicted in FIG. 1 over a system configuration with a single databasethat is receiving concurrent queries and updates from multipleapplications. Furthermore, when applications can operate correctly withless than full serializable isolation (e.g., read-committed), thereplicator module instances in FIG. 1 can be configured to achieve stillhigher system throughput and fault tolerance.

Each application within any node has access to its local database, aswell as access to the databases of other nodes. In general, anapplication is any program or system that can read, add, delete, modify,update, or otherwise manipulate data stored in one or more of thedatabases included in the distributed system. The applications can be,for example, user activated (e.g., banking or accounting software), orautomatically activated by a system (e.g., inventory control softwareprogrammed to run automatically on a daily basis).

The database of each node stores the resources that are used by one ormore of the applications. As is known, a database is a collection ofstructured data. Clients can access the data by submitting transactions,consisting of a set of commands that follow the ACID properties(atomicity, consistency, isolation, and durability). Each site from theserver set maintains a private copy of the database. The initial stateof the database is identical at all servers. A change set defines atransition from the current state of the database to the next state. Theorganization of the databases can be the same (e.g., all relationaldatabases) or different (e.g., some relational databases and some objectoriented databases). Any type or combination of data storage facilitiescan be used, with heterogeneous or homogenous data.

The EVS transport module imposes an ordering on all data items in theglobal model, and can be implemented using conventional total ordertechniques. In one particular embodiment, the EVS transport module isimplemented as described in “From Total Order to Database Replication,”by Yair Amir and Ciprian Tutu. Likewise, the EVS transport module can beimplemented as described in “Extended Virtual Synchrony”, L. E. Moser,Y. Amir, P. M. Melliar-Smith, and D. A. Agarwal. Note that these papersare available online, and are also included in Appendix B of thepreviously incorporated U.S. Provisional Application No. 60/671,373.Further note that an Extended Virtual Synchrony Transport is onespecific implementation of a total order transport module, and thatother embodiments of the present invention can be implemented with totalorder transport techniques other than extended virtual synchronytransports, as will be appreciated in light of this disclosure.

Although each of the replicator module, application(s), and database ofeach site are shown as distinct components in this example embodiment,other configurations can be used as well. For example, any one or all ofthese components can be programmed or otherwise integrated into theother (e.g., the replicator module and EVS transport can be integratedalong with the database onto a server). Conventional or customnetworking and/or inter-process protocols and techniques can be used tocarryout communication between the components in any one site, as wellas between sites. In addition, wired and/or wireless technology can beemployed for intra-site and inter-site communication. For convenience ofillustration, each of the database, replicator module, and EVS transportare depicted as residing on the same machine. In other embodiments, eachmodule or component can execute on a separate machine or a combinationof two or more machines, if so desired.

Replicator Module

FIG. 2 a is a block diagram of a replicator module shown in FIG. 1,configured in accordance with an embodiment of the present invention. Aspreviously explained, the replicator module is communicatively coupledbetween the database and the EVS transport (or other suitable totalorder transport module). Within the replicator are shown modulesrequired to implement the functionality described herein. In particular,this embodiment includes a reconciler module, a change capture retrievermodule, a replication coordination state machine, a fault tolerantdistributed lock manager module, a dynamic membership manager module, atotal persisted order module, and a router. Arrows generally representthe direction of data flow between the modules.

The reconciler module takes as input change sets from the replicationcoordination state machine (RCSM) and updates the data in the databasewith these changes. The reconciler module is essentially more of anadaptor than a state machine, and functions to abstract the databaseimplementation, data schema, data format, and query language specificaspects of reconciliation away from the RCSM (and other modules of thereplicator). The reconciler module input event is ReconcileChangeSet.The RCSM passes a change set to the Reconciler via this event. Thereconciler module outputs database specific update language.

The change capture retriever (CCR) module is used by the RCSM toretrieve committed changes from the local database. The CCR modulereturns change sets to the RCSM. The CCR module abstracts awaydifferences in database structure, schema, and query language from theother components in the replicator. The method of extraction isdependent on the database system, as will be apparent in light of thisdisclosure. A change capture mechanism ensures that changes werepreviously captured during the execution of the transaction and madeavailable to the CCR. In one particular embodiment, there are two inputsto the CCR module: LazyRetrieveChangeSets and ForceRetrieveChangeSets.The RCSM calls the LazyRetrieveChangeSets function when the RCSMrealizes that a transaction has committed (thereby implying that thereare change sets to be retrieved) and there are no waiters for thedistributed locks made available by the commit of this transaction. Thisallows the change set to be asynchronously retrieved, significantlyimproving throughput at the cost of latency. A monotonically increasingunique identifier is associated with each request for change setretrieval. This identifier is used later in an AllChangesRetrieved eventto identify which request has been sent. The RCSM calls theForceRetrieveChangeSets function when the RCSM realizes that atransaction has committed (thereby implying that there are change setsto be retrieved) and there are waiters for the distributed locks freedby the commit of this transaction. Before the locks can be released, allchanges are first sent to the TPOSM. To minimize latency, the CCR moduleensures that the retrieval is done as soon as possible, in oneparticular embodiment. The output of the CCR module in this embodimentis AllChangesRetrieved. After each Retrieve* request, there is at leastone retrieval from the database. After the retrieval is done, the CCRmodule uses the AllChangesRetrieved event to signal the RCSM whichRetrieve* request has been serviced and the change set that wasretrieved. Note that change set retrieval is a significant performancecost. However, a performance optimization can be implemented. Inparticular, observe that a single change set retrieval to the databaseis sufficient to service not just the last Retrieve received by the CCRmodule before the retrieval, but all Retrieve requests received by theCCR module that precede the retrieval. Thus, a single retrieval by theCCR module can result in the servicing of many Retrieve requests and mayresult in many corresponding AllChangesRetrieved events back to theRCSM. The CCR module maintains the invariant that there is at least oneretrieval between the time the Retrieve* was called by the RCSM and theAllChangesRetrieved event is received by the RCSM.

The total persisted order state machine (TPOSM) is used by the RCSM tosend change sets retrieved from the CCR to the other members of thecoordination group. The TPOSM also delivers change sets to the RCSM,whereupon the RCSM will reconcile, if required, the change set using thereconciler module. The TPOSM uses the EVS transport. In one particularembodiment, the TPOSM is based on the method described in the previouslyincorporated “From Total Order to Database Replication” by Amir andTutu, in accordance with one embodiment of the present invention.However, the algorithm described there is augmented and enhanced for thepurposes of implementing the replication methods described herein. Also,various other aspects of the TPOSM were not specified in the Amir andTutu paper. These aspects are described herein. Prior to discussingmodifications to the original TPOSM, some of the original TPOSM is firstdefined (as described by Amir and Tutu). The original TPOSM consisted ofseven states: NonPrimary, Regular Primary, Transitional Primary,Exchange States, Exchange Actions, Construct, Undecided and No. Changesare communicated over a network between members as actions, which areuniquely identified by a monotonically increasing per member indexstarting at 0 and the member unique identifier. The original TPOSMspecified two sets of persistent actions: those in a local ongoing queuethat have not been sent yet, and those in the action list which containsthe total persisted order across all of the members. The original TPOSMdefines three categories of actions: Red, Yellow, and Green. Red actionsare actions whose total persisted order is not known, Yellow actions arethose whose order is known but not confirmed, and Green actions arethose whose order is known and confirmed. An action's color is set bymarking the action; thus, an action is said to have been marked Red orGreen when the accompanying ordering has taken place. A member uniqueidentifier is also referred to herein as the member server ID. Themonotonically increasing index is also referred to herein as the actionindex. An Action ID unique identifies an action and includes a server IDand an action index. Each TPOSM maintains several persistently storeddata structures for managing ordering of actions and coordinatingrecovery. The first of these is the Red Cut, which stores the highestaction index per server ID that the local member has persistently storedin order. The second are the Green Lines, which record the Action ID ofthe last action marked Green for every member. As discussed herein,actions in the action list are also called historical updates. Actionsin the ongoing queue are not historical updates, as they have not beenapplied on other members yet.

In one embodiment of the present invention, the TPOSM defines a peraction member relative total persisted order value. This value is unsetuntil an action has been marked Green at the local member, at which timeit is assigned the next value of the total persisted order. Because theset of members of the coordination group can change with time, the totalpersisted order value on each action is relative to the local member.This value is primarily used in the RCSM for reconciliation purposes.This is a significant departure from the original TPOSM, as thereplication method of this particular embodiment does not execute allupdates in the total order.

In the original TPOSM, new changes from the replica are placed asuncolored actions in the ongoing queue and then communicated to othermembers during Regular Primary and NonPrimary. Because changes in thelocal replica are applied prior to being replicated across thecoordination group, care must be taken when sending actions. When thechanges are received by this member they are removed from the ongoingqueue. In the original TPOSM, upon installation of primary any actionsin the ongoing queue are sent, and upon recovery any actions in theongoing queue are marked Red. The replicator module configured inaccordance with an embodiment of the present invention takes a differentapproach. In this particular embodiment, rather than maintaining anongoing queue, local actions are marked Red immediately. This leads toincreased performance during recovery as well as a more deterministictotal order as such actions are marked Green across all members uponinstallation of primary, rather than in batches at beginning of primary.There is also a performance improvement in maintaining a single list ofactions, because less action copying is needed.

In the original TPOSM the notion of a most up to date member wasintroduced but not defined. Such member is responsible forretransmitting the action list of Green actions in the totally persistedorder during the Exchange Actions state. Because in an arbitrary set ofmembers each member may have different amounts of historical data, thealgorithm for determining the most up to date member relies on theunique action identifiers rather than any member relative index. Thereplicator module configured in accordance with an embodiment of thepresent invention uses the following algorithm to determine the most upto date member. For each member of the coordination group, find thehighest action marked Green across all the members in the configuration.Then for each member of the coordination group, form a set of members ofthe configuration that have marked the highest action Green. Thenintersect those sets, and take the member with the maximal unique memberID.

In the original TPOSM, a retransmission protocol was not well defined.The most up to date member retransmitted the Green actions but it isunclear what the other members are to retransmit. The replicator moduleconfigured in accordance with an embodiment of the present inventionspecifies the protocol as the most up to date member retransmits Greenactions then Red actions, and the other members retransmit Red actionsin unique identifier order. Additionally, because the communicationsystem is not considered an infinite sink and has limited resources, thereplicator module of this particular embodiment specifies a flow controlsystem based on the durable memory size of the actions that have beensent. This has two effects: first it prevents the communication systemfrom being overwhelmed, and secondly it prevents remote out of datemembers from appearing to crash during recovery. Remote data members canappear to crash when the population of Green actions in their actionlists and the application of those actions to their local replicas istime consuming. In this case, the communication system can erroneouslyconclude that those members have crashed, and evict them from theconfiguration thereby producing a configuration change. Because aconfiguration change involves restarting a recovery process or statemachine, configuration changes are expensive, and avoiding them improvesoverall recovery performance.

The original TPOSM removes historical messages that are not neededbecause all members have them. The replicator module configured inaccordance with an embodiment of the present invention implements thisremoval as a process that runs when the action list reaches a multipleof a configurable value, called the White Line limit. Each member candetermine the last action marked Green on other members through theGreen Line information in each action. If a member in the currentconfiguration does not send any actions during long running primariessuch a member can inhibit the removal of historical messages becauseother members won't know the last message that such a member has markedGreen. Therefore, the replicator module of this particular embodiment isprogrammed or otherwise configured to send a status action at afrequency of half of the White Line limit if the local member has notsent any messages. This allows members to continue to remove historicalinformation, thereby preventing the durable storage associated with thereplicator module to grow without bound. In one such embodiment, thereplicator module uses the following algorithm to determine how muchhistorical data may be removed because it is no longer needed: for eachmember of the coordination group, determine the last action it hasmarked Green, and call this the least set. Find the action in the leastset with the lowest total order value in the local action list. The lastunneeded historical action is the action just prior to that least actionwith the lowest total order value in the local action list. All actionsin the local action list prior to the last unneeded action may bediscarded.

The TPOSM receives SendMessage as an input. SendMessage is used by theRCSM to send a change set to deliver to all other RCSM instances in thecoordination group. The message is delivered guaranteeing totalpersisted order. The call is made in the context of a transaction and ismade in the same transaction as the change retrieval executed by the CCRmodule. The TPOSM outputs OnMessage, which is used to notify the RCSMthat a total persisted order message is ready (marked Green) to bereconciled. The message is delivered in the context of a transaction andprovides both the message itself and a unique sequence number.

The fault tolerant distributed lock manager (FTDLM) enables faulttolerant replication, as will be appreciated in light of thisdisclosure. In one particular embodiment, the FTDLM takes as inputnotifications of write locks, transaction begins, and transactioncommits. Here, notifications are directly from the applications. Inother embodiments, these notifications can come from a detector so thatthe applications need not call the FTDLM directly. The FTDLM acquiresand releases distributed locks. The FTDLM uses the same EVS transport asthe TPOSM and sends messages in the same total order stream as theTPOSM. In one embodiment, the FTDLM is generally implemented asdescribed in the previously incorporated U.S. application Ser. No.11/270,196, except that the output is augmented with a notificationLockTransferRequired and an input with ReleaseLockGranted. Theseadditions to the FTDLM can be used to implement a Commit Rule asdescribed in the previously incorporated U.S. application Ser. No.11/292,055. This Commit Rule states that the lock release message issent after the associated changes are sent via the TPOSM. In such anembodiment, the RCSM notifies the FTDLM when a commit has occurred andlocks can be released via an OnXactCompleted event. The FTDLM ismodified to provide an event to the RCSM to notify when a distributedlock transfer is required (e.g., ReleaseLockRequested). In particular,the FTDLM is modified by changing the logic in the lock queue statemachine, as discussed with reference to FIGS. 3a and 3b in thepreviously incorporated U.S. application Ser. No. 11/292,055. The RCSMwill notify the CCR module to retrieve the changes and send them. Thenthe RCSM will notify the FTDLM that it can now proceed with transferringthe lock ownership (e.g., ProceedWithLockRelease).

Inputs of the FTDLM of one embodiment are as follows. The RCSM uses anOnBeginXact event to notify the FTDLM to prepare for the acquisition ofa set of locks. The RCSM uses an OnLock event to notify the FTDLM toacquire a lock on a resource. The resource ID is included in the event.The RCSM uses the OnXactCompleted event to notify the RCSM that thecurrent transaction is completed. The RCSM uses theProceedWithLockRelease event to notify the FTDLM that it is okay toproceed with lock release on a resource. The output of the FTDLMincludes RelaseLockRequested. This event is used to inform the RCSM thatthe FTDLM needs to release a lock, allowing the RCSM to arrange to firstsend all changes before the FTDLM sends a lock release message.

The replication coordinator state machine (RCSM) coordinates theoperation of the CCR module, TPOSM, reconciler module, DMM, and FTDLM toimplement the Read, Write, Commit, and Reconciliation Rules of thereplication techniques as described herein.

The dynamic membership manager (DMM) is configured to manage thepersistent membership of a coordination group. As previously explained,the persistent membership can be, for example, a durable set of serversthat can replicate changes amongst themselves. Changes from servers notin the persistent membership are discarded. The set of recovered membersis a subset of the persistent membership. The persistent membershipchanges incrementally by either removing or adding members. Failedmembers may still be part of the persistent membership. Examplefunctionality and implementation details of the DMM module will bediscussed in further detail with reference to FIGS. 2 b and 3-7.

With regard to the router module, both the FTDLM and the TPOSM use thesame EVS transport (and the same total order group) to send and receivemessages. The router module is used to multiplex (or otherwisecommunicate) input and outputs from the two components into the same EVStransport.

Each of the reconciler, CCR, RCSM, FTDLM, and TPOSM, as well as dataflows therebetween, are discussed in further detail in the previouslyincorporated U.S. application Ser. Nos. 11/292,055 and 11/270,196.

In one particular embodiment, the replicator module is implemented withsoftware (e.g., one or more set of instructions executing on one or moreprocessors or encoded on one or more processor readable mediums).Alternatively, the replicator module can be implemented, for example, inhardware (e.g., gate-level logic) or a combination of hardware andsoftware (e.g., microcontroller with a number of embedded routines forcarrying out the functionality described herein). In addition, note thatfunctionality of one module may be integrated into one or more othermodules in various alternative embodiments. Also, note that the DMMfunctionality can be implemented without the FTDLM functionality (i.e.,the dynamic membership management techniques described herein can beperformed without fault tolerance and/or without lock management).

Dynamic Membership Manager

FIG. 2 b is a block diagram of a dynamic membership manager (DMM) shownin FIG. 2 a, configured in accordance with an embodiment of the presentinvention.

As can be seen, the DMM includes an entrant state machine, a registrarstate machine, a registrar database initialization state machine, anentrant database initialization state machine, and a membership leavestate machine. A network interface is also provided that allows the DMMto communicate with other sites on the network. FIG. 2 b furtherdemonstrates data flow between the DMM and the RCSM and FTDLM, anddepicts events that the various modules use to interact with each otherto implement DMM functionality.

Prior to discussing each state machine in detail, it may be helpful toidentify some general terms and assumptions to facilitate understandingof dynamic membership management. Assume for purposes of this discussionthat a server (or other site) includes a replicator module, an instanceof the FTDLM, and a replicated database (note, however, that thereplicator module can operate without an FTDLM if so desired). A serveralso has a server ID. Further assume that the replicator module includesdurable storage for an action list, an ongoing queue, the Red Cut andGreen Lines, and persistent quorum information hereafter referred to asthe replicator durable storage or data. Further assume that a persistentmembership includes a durable representation of the set of server IDs ofservers that are allowed to replicate databases and a group name. Onlyservers in the persistent membership may replicate databases; twodifferent persistent memberships with different group names must nothave any servers in common, in accordance with this particular example.The DMM uses an additional state in the RCSM: Joining Non Primary, whichbehaves similarly to Non Primary except that the RCSM is instructed todisconnect from the EVS transport and stay disconnected until instructedto reconnect. In Joining Non Primary, the RCSM does not receive viewchanges; the RCSM stays in Joining Non Primary until the DMM instructsthe RCSM to leave Joining Non Primary by reconnecting to the EVStransport. A member is a server that is included in the persistentmembership. A member may have failed or recovered. Assume that amember's durable storage includes identifying information for everyother member such as member names and server ID. An entrant is a serverseeking to become a member. A registrar is a member of the persistentmembership which helps the entrant attain membership by modifying thepersistent membership on the entrant's behalf and initializing theentrant's database. In the event that the registrar fails, anothermember of the persistent membership may replace the failed member asregistrar for the entrant. A Membership Request message is sent by anentrant to a registrar and includes the entrant's server ID, a name forthe entrant, and other administrative information. A Membership Changemessage is an action which modifies the persistent membership. AMembership Change message behaves like any other action, and is appliedin a totally persisted order at all members. A Membership Change messagemay either add a single server, or remove a single member. A registrarmodifies the persistent membership to add the entrant by sending aMembership Change message that adds the entrant to the persistentmembership. The registrar and entrant then initialize the entrant'sreplica. Should the registrar fail, the entrant may reconnect to anothermember and continue initializing the replica. Techniques by which thereplica can be initialized will be discussed in turn. A member may leavethe persistent membership by sending a Membership Change message.Likewise, an arbitrary member A may be evicted by another member B if Bsends a Membership Change message on behalf of A.

The state machine diagrams shown in FIG. 2 b (as well as other statemachine diagrams herein) use the following conventions: each boxrepresents a state of the state diagram; each arrow has an Event and anAction labeled with “E:” or “A:” respectively; events trigger thetransition as well as the action in the arrow; and events that are notexplicitly depicted in the diagrams are either not possible, or areignored. Additional details of each state machine are further discussedwith reference to corresponding FIGS. 3-7.

Of the five state machines included in the DMM of this embodiment, theentrant state machine, the registrar state machine, the entrant databaseinitialization state machine, and the registrar database initializationstate machine are directly involved in the membership join protocol. Themembership leave state machine is utilized when a member wishes tovoluntarily leave the persistent membership. Note, however, that anevicted member does not use the membership leave state machine. Theregistrar and entrant state machines handle the mechanics of adding theentrant to the persistent membership and populating the entrant'sreplicator durable storage. The registrar and entrant databaseinitialization state machines are concerned specifically with theinitialization of the entrant's database.

The network interface of this embodiment (TCPConnection) is a TCP/IPnetwork interface, although any network interface distinct from the EVStransport is suitable.

FIG. 2 b is best explained in terms of operational tasks. Arrows denotedata and control flow. The DMM has three primary tasks: acting as aregistrar, acting as an entrant, and removing the local member from thepersistent membership. When the DMM is acting as a registrar, theregistrar state machine receives incoming Connections and MembershipRequests from the network interface. Upon receipt of such requests, theregistrar state machine Requests a Membership Lock of the FTDLM, whichreports that the Membership Lock is Acquired when it has obtained a lockon the entrant's name. The registrar state machine then Requests aMembership Change of the RCSM, which reports the Membership Change whenthe request has been marked Green and the membership has changed. Thenthe registrar state machine asks the FTDLM to release locks byRequesting Locks Release. When Locks are Released by the FTDLM, theregistrar state machine then Requests Transition to Joining Non Primaryof the RCSM. When the RCSM has Entered Joining Non Primary, theregistrar state machine knows that the replicated database will stayconstant, so the registrar state machine sends a Membership Grant to theentrant state machine (via the network interface) and Starts DBI(Database Initialization) by engaging the registrar databaseinitialization state machine. When DBI is Finished (as indicated by theregistrar database initialization state machine), the registrar statemachine sends Historical Updates and Replicator State to the entrantstate machine, and then Resets.

When the DMM receives a Join request, it constructs or otherwise createsan entrant state machine. The entrant state machine sends a MembershipRequest to the registrar state machine (via the network interface). Uponreceipt of a Membership Grant, the entrant state machine Starts DBI toinitialize the entrant database (by engaging the entrant databaseinitialization state machine). When DBI is Finished, the entrant statemachine receives Historical Updates and Replicator State from theregistrar state machine (via the network interface). Upon installationof replicator persistent state, the entrant state machine constructs theRCSM and joins the EVS transport group.

When the DMM receives a request to Leave the persistent membership, theDMM constructs a membership leave state machine, which first RequestsException to all Transactions of the FTDLM, and then Requests aMembership Change of the RCSM. When the Membership Change occurs (asindicated by the RCSM), the membership leave state machine RequestsTransition to Non Primary of the RCSM. Upon Entering Non Primary (asindicated by the RCSM), the membership leave state machine is complete;the RCSM is halted and the FTDLM stops processing. In addition, themembership leave state machine instructs the RCSM to disconnect from theEVS transport, thereby preventing the RCSM from receiving view changesand leaving Non Primary.

Registrar State Machine

FIG. 3 illustrates a registrar state machine of the dynamic membershipmanager shown in FIGS. 2 a and 2 b, configured in accordance with anembodiment of the present invention.

The purpose of the registrar state machine is to help entrants becomemembers. For servers which are already members, the registrar statemachine is created upon initialization of the replicator module. Forservers which have just become members (either by following themembership join protocol or by starting a new coordination group), theregistrar state machine is created upon entering forward processing. Inthe latter case of starting a new coordination group, the creation ofthe registrar state machine is delayed until the replicator module hasbrought the replicated database and the persistent membership up todate.

The state definitions of the registrar state machine in this exampleembodiment are as follows:

IDLE: The registrar state machine is waiting for an entrant to connect.

CONNECTED: The registrar state machine is connected to an entrant, buthas not received the entrant's server ID or name.

WAITING FOR LOCK: The registrar state machine is waiting for themembership lock (see also AcquireLock function definition).

WAITING FOR MEMBERSHIP CHANGE: The registrar state machine is waitingfor the membership change to be given a totally persisted order andapplied to the persistent membership.

WAITING TO RELEASE LOCKS: The registrar state machine is waiting for theFTDLM to release locks.

WAITING TO ENTER JOINING NON PRIMARY: The registrar state machine iswaiting for the replicator module to enter Joining Non Primary.

DATABASE INITIALIZATION: The registrar database Initialization statemachine operates in this state. Operation of the registrar state machineis suspended while the registrar database initialization state machineis processing, and resumes once the registrar database initializationstate machine has completed operation.

SEND HISTORICAL: In this state, the registrar state machine is sendinghistorical updates (see also the HistoricalUpdates function definition).

The event definitions of the registrar state machine in this exampleembodiment are as follows:

OnConnection: This event is signaled when an entrant makes a connectionto the registrar state machine.

OnReceiveServerIDAndName: This event occurs when the registrar statemachine receives the server ID, name, and weight of the entrant.

OnCreate: This is the initial event, and indicates that the registrarstate machine has been created.

OnConnectionError: This event is signaled when either the entrant hassent an error message, or the connection to the entrant has been lost.

OnLockAcquired: This event is signaled when the member name lock hasbeen acquired.

OnReceiveMembershipGrant: This event is signaled when the MembershipChange Message has been assigned a total persisted order, persistentlystored, and the Persistent Membership has changed.

OnLeaveInProcess: This event is signaled when the FTDLM leaves forwardprocessing.

OnLocksReleased: This event is signaled when the FTDLM has sent allchanges and released all locks.

OnEnterNonPrimary: This event is signaled when the replicator module hasentered Joining Non Primary. Joining Non Primary is a regular NonPrimary state, except that EVS transport changes do not change thereplicator module state.

OnFinishDBI: This event is sent by the registrar database initializationstate machine indicating successful completion.

OnDBIError: This event is sent by the registrar database initializationstate machine indicating unsuccessful completion.

OnReceiveAck: This event is signaled when the Entrant has sent anacknowledgement during the historical update protocol.

OnFinishSendingHistorical: This event is signaled when historicalupdates are sent.

The function definitions of the registrar state machine in this exampleembodiment are as follows:

Initialize( ): Initialization is comprised of initializing the networkconnection and starting to listen for entrant requests.

Reset( ): Reset destroys any memory of the last entrant's information.If the registrar state machine requested that the replicator moduleleave Primary and enter Joining Non Primary, then Reset instructs thereplicator module to reenter Primary. The replicator module does this byreconnecting to the EVS transport.

Error( ): If the error is caused by the entrant state machine (via anOnConnectionError event), then Error just calls Reset. If any othererror occurred, Error sends an error message to the entrant statemachine and then calls Reset.

AcquireLock( ): This function instructs the FTDLM to acquire the locknamed GroupName:MemberName, where GroupName denotes the name of thegroup that the registrar state machine is part of, and MemberNamedenotes the name sent by the entrant state machine.

ReleaseLocks( ): This function requests that the FTDLM release allcurrent locks. The FTDLM exceptions all transactions in process, causesthe RCSM to send all accumulated unpropagated updates, and sends lockrelease messages.

makePersMemReq( ): This function requests that the replicator modulesend a Membership Change message adding the entrant to the membership.The message is sent over the EVS transport, thereby propagating the(logical) update to the persistent membership.

RequestJoiningNonPrimary( ): This function requests that the replicatormodule enter Joining Non Primary.

StartDBI: This function sends an event to the registrar databaseinitialization state machine indicating that it can proceed. Thisfunction also suspends operation of the registrar state machine untilthe registrar database initialization state machine has finishedoperating.

SendHistorical: This function sends historical updates. The registrarstate machine sends a subset of the updates that have been assigned atotally persisted order. The first update that the registrar statemachine sends is determined by the database initialization mode of theentrant state machine. If the entrant state machine has requestedbackup/historical initialization mode, then the first update the entrantstate machine does not have becomes the first update the registrar statemachine sends; elsewise the registrar state machine sends its entirehistorical update list. The registrar state machine and entrant statemachine implement a flow control system based on size of updates: whenthe size of accumulated sent updates exceeds a certain (e.g.,configurable) limit, the registrar state machine sends anacknowledgement request and waits for an OnReceiveAck event.

SendMoreHistorical: Upon receipt of an acknowledgement from the entrantstate machine, the registrar state machine continues sending morehistorical updates, picking up where it left off.

SendRMState: This function causes the registrar state machine to sendits current persistent replicator module state, including the persistentmembership. The sent Red Cut is set to be a duplicate of the sent GreenLines, since only totally persisted updates are sent to the entrantstate machine.

InMembership(Entrant): This function returns true if the entrant'sserver ID is in the active persistent membership; otherwise, thisfunction returns false.

Duplicate(Name): This function returns true if the entrant's name isalready in use in the active or inactive persistent membership;otherwise, this function returns false.

InvalidEntrantDBIMode( ): This function returns true if the entrantstate machine has requested backup/historical mode and the last appliedupdate in the entrant's database is not found in the registrar'shistorical update list.

This particular embodiment of the registrar state machine includes useof the fault tolerant distributed lock manager (FTDLM) in order toensure uniqueness of server names within the persistent membership. Theregistrar state machine and dynamic membership manager (DMM) in generalmay be utilized in the absence of the FTDLM by omitting states WAITINGTO RELEASE LOCKS and WAITING FOR LOCK, if so desired. Note, however,that in the event that these states are omitted, the WAITING FORMEMBERSHIP CHANGE state still exists, as this state is necessary inorder to initialize the entrant's database after the entrant has beenadded to the persistent membership.

Entrant Database Initialization State Machine

FIG. 4 illustrates an entrant database initialization state machine ofthe dynamic membership manager shown in FIGS. 2 a and 2 b, configured inaccordance with an embodiment of the present invention.

The entrant database initialization state machine operates when theentrant state machine is in the database initialization state. Thepurpose of the entrant database initialization state machine is toinitialize the entrant's replicated database. If entrant initializationis in verification/update mode, then an additional role for databaseinitialization is to create the necessary updates in the entrant'saction buffer that would introduce data only on the entrant's databaseto the rest of the replication group. These updates are created as Redactions; upon joining the EVS transport group, the replicator modules ofall connected members will apply these updates and thus make theentrant's new data available. Members not connected at the time theentrant joins the EVS transport group will eventually receive theseupdates in the normal manner during recovery.

When in snapshot or update/verify mode, processing of the entrantdatabase occurs by partitioning the database and processing eachpartition in turn. These partitions can be the same as those utilized bythe FTDLM as locks, but do not have to be. Any suitable partitioningmethod can be used as long as both the registrar and entrant statemachines use the same method.

As with the processing of historical updates, the registrar and entrantdatabase initialization state machines implement flow control in orderto avoid dropped connections due to buffer overflow. Entrant tasksduring database initialization can be time consuming, and if theregistrar is allowed to send data without flow control the registrar caneasily overflow network buffers while the entrant will be busy withdatabase access. Network buffer overflow may result in connectionerrors, which may in turn cause other events (e.g., the registrar statemachine, the entrant state machine, the registrar databaseinitialization state machine, and the entrant database initializationstate machine to restart).

As will be apparent in light of this disclosure, the entrant databaseinitialization state machine can be implemented as multiple statemachines (e.g., one for each database initialization mode). However,because of functional overlap and for brevity, the functionality hasbeen collapsed into a single state machine.

The state definitions of the entrant database initialization statemachine in this example embodiment are as follows:

INITIALIZE: In the INITIALIZE state, the entrant database initializationstate machine determines the mode of database initialization to perform,acquires database connections, and other such initializationfunctionality.

DELETE DATA: In the DELETE DATA state, the entrant databaseinitialization state machine deletes all existing data in the replicateddatabase.

APPLY DATA: In the APPLY DATA state, the entrant database initializationstate machine applies data sent from the registrar databaseinitialization state machine to the replicated database. If the databasewas emptied via DELETE data, then application includes creating orinserting new data. If the database was not emptied, then applicationmay include updating existing data or creating new data.

VERIFY AND UPDATE: In the VERIFY AND UPDATE state, the entrant databaseinitialization state machine verifies data partitions in turn. If a datapartition is determined to contain different data than the registrar'scopy, then the registrar database initialization state machine sends anymissing or different data, which the entrant database initializationstate machine then applies in the APPLY DATA state. If, afterapplication of data from the registrar database initialization statemachine, data partitions still contain differences, the data that existsonly on the entrant's replicated database is turned into updates in theentrant's action buffer with the CreateUpdates function (which will bediscussed in turn).

ERROR: The ERROR state signifies that some error occurred duringdatabase initialization. When the entrant database initialization statemachine enters the Error state, the entrant state machine is sent anOnDBIError event (as will be discussed with reference to FIG. 6).

DONE: The DONE state is the final state of the entrant databaseinitialization state machine. When the entrant database initializationstate machine enters the DONE state, the entrant state machine is sentan OnDBIFinished event (as will be discussed with reference to FIG. 6).

The event definitions of the entrant database initialization statemachine in this example embodiment are as follows:

StartDBI: This event is sent by the entrant state machine when it movesto the DATABASE INITIALIZATION state.

OnSnapshotDBI: This is a pseudo event, which returns true when theentrant database initialization state machine is in snapshotinitialization mode.

OnHistoricalDBI: This is a pseudo event, which returns true when theentrant database initialization state machine is in backup/historicalinitialization mode. In this mode, the entrant database initializationstate machine does nothing.

OnVerifyDBI: This is a pseudo event, which returns true when the entrantdatabase initialization state machine is in verify/update mode.

OnDataDeleted: This event signifies that all data in the entrant'sreplicated database has been deleted.

OnDataDeleteError: This event signifies that not all data in theentrant's replicated database could be deleted.

OnDataApplicationError: If the entrant is in snapshot mode, this eventsignifies that not all data could be inserted into the entrant'sreplicated database. If the entrant database initialization statemachine is in update/verify mode, then this event signifies that not alldata could be updated or inserted into the entrant's replicateddatabase.

OnDataAvailable: This event signifies that the entrant databaseinitialization state machine has received data from the registrardatabase initialization state machine.

OnDataDone: When the registrar database initialization state machine hasfinished sending data, this event is signaled.

OnVerificationError: When the entrant database initialization statemachine detects an unrecoverable verification error, this event issignaled. An unrecoverable verification error may occur, for example,when the entrant database initialization state machine is unable toreconcile differences (e.g., by applying inserts, updates, or creatinghistorical updates) between the registrar's replicated database and theentrant's replicated database.

OnUpdateCreationError: This event is signaled when the entrant databaseinitialization state machine cannot create the necessary updates topropagate data that is new on the entrant's replicated database.

OnReceiveAckRequest: This event signifies that the registrar databaseinitialization state machine sent an acknowledgement request.

OnVerifyDifference: This event is signaled when the entrant databaseinitialization state machine detects differences in the entrant's andregistrar's replicated databases.

OnVerifyCompleted: This event is signaled when verification is complete.

OnUpdatesCreated: This event is signaled when creation of updates iscomplete.

The function definitions of the entrant database initialization statemachine in this example embodiment are as follows:

Initialize: The initialize function determines which mode databaseinitialization is in. If in snapshot mode, then dependencies betweendata partitions are computed and initialization occurs in dependencyorder.

Finish: The Finish function signals the entrant state machine thatdatabase initialization was successful.

DeleteData: The DeleteData function deletes existing data from theentrant's replicated database in reverse dependency order.

StartVerify: The StartVerify function initializes the verification ofdata partitions. Verification occurs in any method which ensures thatthe data in the registrar and entrant databases for a specific partitionis identical. This can occur, for example, via checksumming or directcomparison. Each data partition is verified in turn.

Error: When an error is detected, the entrant database initializationstate machine sends a DBIError event to the entrant state machine. TheEntrant database initialization state machine also sends an error to theregistrar database initialization state machine.

SendAck: This function sends an acknowledgement to the registrardatabase initialization state machine.

WaitForData: This function sends a data request to the registrardatabase initialization state machine and waits for a response.

ApplyData: This function alters the entrant's replicated database bysome combination of inserting data and updating existing data. Data fromthe registrar database initialization state machine may be inserted intothe entrant's replicated database, and data in the entrant's databasemay be updated to be identical to that in the registrar's replicateddatabase.

CreateUpdates: When verification is complete the entrant determineswhich data partitions in the entrant's replicated database contain datathat the registrar's replicated database does not contain. The data inthese data partitions is then used to create update actions which aremarked Red. These actions consist of the necessary inserts to populate areplicated database such that upon reconciliation of the actions thereplicated database contains the same data as that in the entrant'sreplicated database.

Registrar Database Initialization State Machine

FIG. 5 illustrates a registrar database initialization state machine ofthe dynamic membership manager shown in FIGS. 2 a and 2 b, configured inaccordance with an embodiment of the present invention.

The registrar and entrant database initialization state machines can bethought of as two cooperating state machines, in that each operates withthe help of the other. Whereas the entrant database initialization statemachine is bringing the entrant's replicated database up to date, theregistrar database initialization state machine is simply providing datafrom its replicated database, which is, by definition, already up todate.

As with the entrant database initialization state machines, theregistrar database initialization state machine utilizes flow control toensure that the registrar database initialization state machine does notsend data more quickly than the entrant state machine can process it.When the entrant database initialization state machine requests data fora data partition, it may not know how large that partition is; thereforethe registrar database initialization state machine sends some data andthen initiates an acknowledgement request cycle. When the entrantdatabase initialization state machine acknowledges the data, theregistrar database initialization state machine continues sending data.

The registrar database initialization state machine partitions thedatabase in the same manner as the entrant database initialization statemachine. As with the entrant database initialization state machine, onelogical state machine can be implemented for each entrant databaseinitialization mode (of which there are three; others will be apparentin light of this disclosure). However, these three state machines havebeen collapsed into one state machine as shown in FIG. 5 due tofunctional overlap and brevity.

The state definitions of the registrar database initialization statemachine in this example embodiment are as follows:

INITIALIZE: In the INITIALIZE state, the entrant's databaseinitialization mode is determined, data partitions are determined,database connections are obtained, and any other necessaryinitialization occurs.

SEND DATA: When in SEND DATA state, the registrar databaseinitialization state machine is extracting data from a data partition inthe registrar's replicated database and sending it to the entrantdatabase initialization state machine.

VERIFY: In VERIFY mode, the registrar database initialization statemachine is aiding the entrant database initialization state machine inverification of a particular data partition. In one embodiment, this iscarried out via direct comparison between the contents of theregistrar's database and the entrant's database.

ERROR: The ERROR state indicates that some failure occurred; either aconnection error, a local data access error, or an error on the entrantdatabase initialization state machine.

DONE: The DONE state indicates that database initialization completedsuccessfully.

The event definitions of the registrar database initialization statemachine in this example embodiment are as follows:

StartDBI: This event is sent by the registrar state machine whendatabase initialization should take place.

OnSnapshotDBI: This is a pseudo event, which returns true when theentrant database initialization state machine is in snapshotinitialization mode.

OnHistoricalDBI: This is a pseudo event, which returns true when theentrant database initialization state machine is in backup/historicalinitialization mode. In this mode, the registrar database initializationstate machine does nothing.

OnVerifyDBI: This is pseudo event, which returns true when the entrantdatabase initialization state machine is in verify/update mode.

OnFinishedSendingData: This event is signaled when the registrardatabase initialization state machine has sent all the datacorresponding to a specific request.

OnFinishedData: This event is signaled when the registrar databaseinitialization state machine has no more data to send.

OnConnectionError: This event signifies that some sort of network erroroccurred, or that the entrant database initialization state machine sentan error message other than a verification error message.

OnDataError: This event is signaled when the registrar databaseinitialization state machine cannot obtain data for a particular datapartition for which data exists from the database.

OnAck: This event is signaled when the registrar database initializationstate machine receives an acknowledgement from the entrant databaseinitialization state machine.

OnVerificationError: This event is signaled upon receipt of a messageindicating that the entrant database initialization state machine hasdetermined that verification has failed

OnFinishVerify: This event is signaled upon receipt of a messageindicating that the entrant database initialization state machine hasfinished verification.

OnDataRequest: This event is signaled upon receipt of a request for datafor a particular partition from the entrant database initializationstate machine.

The function definitions of the registrar database initialization statemachine in this example embodiment are as follows.

Initialize: The initialize function initializes the registrar databaseinitialization state machine, acquires database connections, ensuresthat the database can be read, and other such initializationfunctionality.

SendData: The SendData function sends the data contained in therequested data partition. If the data partition contains more data thancan be sent without an acknowledgement request, then the SendDatafunction follows the data with an acknowledgement request.

SendMoreData: Upon receipt of an acknowledgement, the SendMoreDatafunction continues sending data from the current data partition.

Finish: When there is no more data to send, and the entrant databaseinitialization state machine signals that verification or snapshottingwas successful, the Finish function sends a DBIFinished event to theregistrar state machine.

WaitForVerifyRequest: Verification occurs by data partition in turn. Theentrant database initialization state machine verifies the firstpartition, and the second and so on. Verification of each data partitionresults in a request for data. The WaitForVerifyRequest function waitsfor such requests.

Error: The Error function signals that verification failed by sending aDBIError event to the registrar state machine.

Entrant State Machine

FIG. 6 illustrates an entrant state machine of the dynamic membershipmanager shown in FIGS. 2 a and 2 b, configured in accordance with anembodiment of the present invention.

The purpose of the entrant state machine is primarily to receive data ina specified order from the registrar state machine. The entrant statemachine commences upon a client request to join the persistentmembership. When the entrant's request to join has been granted, theentrant state machine and registrar state machine arrange for thedatabase initialization protocol to operate. This causes the registrardatabase initialization state machine and entrant databaseinitialization state machine to operate. When database initialization iscomplete, the registrar state machine sends any historical updates thatthe entrant state machine does not have in total persisted order, andthe entrant state machine stores those updates in its action list,optionally applying them if in backup/historical mode. Finally, theregistrar state machine sends a modified copy of the registrar'sreplicator module persistent state. This copy is identical to thatstored on the registrar state machine except for the Red Cut, which isset to be a duplicate of the Green Lines since the entrant state machinedoes not have any Red actions. Of course, if the entrant state machinecreated Red actions as part of database initialization, the Red Cut forthose actions is preserved.

An entrant state machine may use more than one registrar state machineif the membership join protocol fails in some way. The entrant statemachine keeps a transient list of available registrars; as each attemptfails the entrant state machine determines whether to remove the currentregistrar from the list; then the entrant state machine tries the nextregistrar. When the entrant state machine reaches the end of theregistrar list, it starts at the beginning again. If there are noavailable registrars, then the membership join protocol fails.

The state definitions of the entrant state machine in this exampleembodiment are as follows.

WAIT FOR MEMBERSHIP GRANT: In this state, the entrant state machine hassent a Membership Request message to the current registrar state machineand is waiting for a grant.

DATABASE INITIALIZATION: The entrant state machine is in this state whenthe entrant database initialization state machine is operating.

RECEIVE HISTORICAL UPDATES: The entrant state machine is receivinghistorical updates. If the entrant database initialization state machineis in backup/historical mode, the entrant state machine is also applyingthese updates.

RECEIVE REPLICATOR STATE: The entrant state machine is receiving thepersistent replicator module's state.

ERROR: When the entrant state machine runs out of registrars, it entersthe Error state. No further processing occurs.

DONE: When the entrant state machine has successfully completed, itenters the DONE state. Upon entering the DONE state, the replicatormodule and fault tolerant distributed lock manager (FTDLM) are created.

The event definitions of the entrant state machine in this exampleembodiment are as follows.

EntrantCreated: This event is signaled when the DMM receives a requestto join the persistent membership from a client. The DMM does notinitiate the join until a client specifically requests that this serverbecome a member and presents a list of registrars.

OnMembershipGrant: This event is signaled when the entrant state machinereceives a Membership Grant message from the registrar state machine.

OnDBIError: This event is sent by the entrant database initializer statemachine when it encounters an error.

OnDBIFinished: This event is sent by the entrant database initializerstate machine when it finishes successfully.

OnAckRequest: This event is signaled when the entrant state machinereceives an acknowledgement request from the registrar state machineduring reception of historical updates.

OnHistoricalUpdatesFinished: This event is signaled when the entrantstate machine has populated the action list and applied all updates ifin backup/historical mode.

OnHistoricalError: This event is signaled when the entrant state machineis either unable to populate the action list with historical updates, oris unable to apply those historical updates if in backup/historicalmode.

OnRMStateFinished: This event is signaled when the entrant state machinehas installed the persistent replicator module state.

OnRMStateError: This event is signaled when the entrant state machinecannot install the persistent replicator module state.

OnConnectionError: This event is signaled when the connection to theregistrar state machine is dropped, or when the entrant state machinereceives an error message from the registrar state machine.

The function definitions of the entrant state machine in this exampleembodiment are as follows.

MoreRegistrars: The entrant state machine keeps a transient list ofavailable registrars. This function returns true when this list is notempty.

Reset: This function is called upon error. Depending on the type oferror, the current Registrar may be removed from the list of availableregistrars. Then, the entrant state machine sends a Membership Requestmessage to the next available registrar state machine.

StartDBI: This function starts the entrant database initializer statemachine.

SendAck: This function sends an acknowledgement to the registrar statemachine when the entrant state machine receives an acknowledgementrequest.

Error: This function signals an error to the dynamic membership manager,signifying that the membership join protocol failed.

Finish: The Finish function signals success to the entrant statemachine's caller, signifying that the membership join protocolsucceeded.

WaitForRMState: This function waits until the registrar state machinesends the replicator module's persistent state.

Membership Leave State Machine

FIG. 7 illustrates a membership leave state machine of the dynamicmembership manager shown in FIGS. 2 a and 2 b, configured in accordancewith an embodiment of the present invention.

The purpose of the membership leave state machine is to coordinate theshutdown of various components upon voluntary leave. The membershipleave state machine is only used during voluntary leave, which can onlybe initiated via client request.

The state definitions of the membership leave state machine in thisexample embodiment are as follows.

INITIALIZE: The INITIALIZE state determines whether this member canprocess a Leave request.

ERROR: When the ERROR state is reached, an exception is thrown.

WAIT FOR MEMBERSHIP CHANGE: In this state the membership leave statemachine is waiting for the Membership Change message containing theLeave for this member to be marked Green.

WAIT FOR NON PRIMARY: In this state the membership leave state machineis waiting for the RCSM to enter non primary.

DONE: When the DONE state is reached, the persistent replicator statehas been set such that upon process failure the member will continue tonot be a member.

The event definitions of the membership leave state machine in thisexample embodiment are as follows.

LeaveRequest: A LeaveRequest event is signaled by the DMM and notifiesthe membership leave state machine that the client wishes this member toleave the membership.

InPrimary: This is a pseudo-event, that returns true when the RCSM is inREGULAR PRIMARY and the FTDLM is forward processing.

NotInPrimary: This is a pseudo-event, that returns true when the RCSM isnot in REGULAR PRIMARY and the FTDLM is not forward processing.

OnLeavePrimary: This event is signaled when the RCSM leaves RegularPrimary.

OnMembershipChange: This event is signaled when the RCSM marks aMembership Change message Green and modifies the persistent membership.

OnEnterNonPrimary: This event is signaled when the RCSM enters NONPRIMARY.

The function definitions of the membership leave state machine in thisexample embodiment are as follows.

Initialize: The Initialize function determines whether the Leave requestcan proceed.

Error: The Error function notifies the client that the Leave failed.

Finish: The Finish function sets the persistent membership state to benot in the persistent membership.

ExceptionAllTransactions: The ExceptionAllTransactions function requeststhe FTDLM to exception all in process transactions.

ShutdownRegistrar: If this member is configured to act as a registrar,then this function halts the registrar state machine so that it nolonger accepts incoming connections.

RequestMembershipChange: This function requests the RCSM to send aMembership Change message requesting that this member leave thepersistent membership.

RequestNonPrimary: This function requests the RCSM to enter NON PRIMARYby disconnecting from the EVS transport.

Membership Join Protocol and Methodology

As will be appreciated in light of this disclosure, membership join isthe process by which an entrant becomes a member. The process includesmodifying the persistent membership to include the new member,initializing the entrant's replica, initializing administrative data,and initializing the entrant's replicator durable storage. In additionto a server ID, each member also has a name (e.g., human readableshorthand), in accordance with one particular embodiment of the presentinvention. When a registrar receives a Membership Request message froman entrant, that registrar determines: (a) whether any other member hasthe same name, and (b) whether some other entrant is attempting toattain membership using the same name. In order to achieve this, theregistrar employs the FTDLM to acquire a distributed lock on theentrant's name. Upon acquisition of this lock, the registrar verifiesthat no other member has the entrant's name. If this is not the case,the entrant is refused membership. In the event that the entrant isalready part of the persistent membership, the registrar does not addthe entrant again. Upon modification of the persistent membership, theregistrar and entrant can initialize the entrant's database.

The replicator module described herein utilizes the services of a groupcommunication system (GCS) which provides an Extended Virtual Synchrony(EVS) service. The GCS manages transport between a set of servers thatare currently communicating with each other. Entrance into this set ofservers is determined by a connection protocol, which forces viewchanges. The replicator module performs recovery upon view changes,which is computationally expensive. In order to avoid view changes, themembership join protocol does not utilize the GCS; instead the registrarand entrant communicate via TCP/IP connection, although any suitablenetwork protocol may be utilized. Note that the registrar and entrantutilize a different communication system from the replicator module soas not to create unnecessary view changes, which result in thecoordination group leaving forward processing and thereby decreasesavailability. Further note the distinction between a coordination group(which includes sites replicating databases) and a GCS group (whichincludes servers communicating, but not necessarily part of the samecoordination group). For instance, a coordination group could includeservers A, B, C and D. Suppose a new server, E, connects to the EVStransport. E is now part of the GCS group. This produces a view change,and the new view includes A, B, C, D, and E. However, E is not part ofthe coordination group, as it is not in the persistent membership. Thus,all the servers can all talk to each other, but E does no replicationand does not send updates. Note the inherent security feature here, inthat updates attempted by malicious users that hack into the GCS groupwill be ignored by the members of the coordination group, because themalicious user is not part of the persistent membership. Additionalsecurity mechanisms can be implemented to prevent malicious access atthe GCS group level as well, if so desired (e.g., intrusion detectionsystems).

As previously referred to in the description of the state machines,there are three methods for database initialization: snapshot,verify/update, and backup/historical. In the snapshot method, theentrant database is initialized via standard database snapshotting. Inthe verify/update method, the contents of the registrar's and entrant'sdatabases are compared, and any inconsistencies are rectified. In thebackup/historical initialization mode, the database is brought up todate from an offline copy by applying historical updates. In order toprovide a consistent snapshot of the replica, the registrar voluntarilydisconnects from the group communication system. This forces the FTDLMto inhibit all further updates, thereby providing a consistent snapshot.

Alternatively, database initialization could occur without the registrardisconnecting from the GCS if the registrar disallowed forwardprocessing and acquired distributed locks on partitions of the databasein turn. However, this may reduce the availability of the replicationnetwork. In particular, if the partitions are too small, this increaseslock message traffic which burdens all servers. On the other hand, ifthe partitions are too large then contention will occur. Rather thanimpede the entire replication group, one choice is to only have theregistrar disconnect.

The Membership Join Protocol configured in accordance with oneembodiment of the present invention includes the steps 1 through 4 below(or a variation thereof), and any one or more of the refinements. Notethat some of the refinements are dependent on earlier refinements, whileothers are independent.

Step 1: The entrant contacts the registrar, which sends a MembershipChange message on the entrant's behalf.

Step 2: Once the Membership Change message has been persisted, totallyordered, and the persistent membership has changed, the registrar andentrant initialize the entrant's database.

Step 3: Should the connection between the registrar and the entrant besevered, the entrant can reconnect to any available registrar.

Step 4: The entrant and registrar join the GCS group and recovers.

Refinement 1: The entrant sends a Membership Request message to theregistrar. When the entrant wishes to utilize backup/historicalinitialization, the Membership Request message also contains an actionID of the last action that was applied to the entrant's database. Thisidentifier contains the server ID of the server that generated theupdate as well as a server relative index. Therefore, the action IDuniquely identifies the containing persistent membership. In the eventthat the registrar does not contain the historical update for thisaction ID, the registrar refuses to proceed.

Refinement 2: The registrar acquires a distributed lock on the group andentrant name utilizing the FTDLM. Upon receipt of the distributed lock,the registrar examines the persistent membership to ensure that no otherserver utilizes the entrant's name. Each server ID is paired withexactly one name. If the entrant's name is already in use by a serverother than the entrant, the registrar refuses to proceed. Acquisition ofthe distributed lock requires forward processing; therefore a registrarcan only service an entrant if the registrar has attained quorum.

Refinement 3: If the entrant is not already part of the persistentmembership, the registrar sends the Membership Change message containingthe entrant's server ID, name, and other identifying information. Whenthe Membership Change message has been marked Green and the persistentmembership has been changed, the registrar proceeds.

Refinement 4: The registrar's FTDLM aborts any transactions in processand releases any locks held by the registrar. Locks are released so thatother members may utilize the locks, and also so that no unpropagatedchanges exist in the registrar's replicated database. The registrar thenleaves the GCS group. This causes the replicator module and the FTDLM tostop forward processing, which allows the registrar to provide aconsistent snapshot to the entrant.

Refinement 5: The registrar and entrant then initialize the entrant'sdatabase.

Refinement 5a: If the entrant is using snapshot initialization, theentrant's database is initialized with the contents of the registrar'sdatabase. These contents are guaranteed not to change while the entrantand registrar are communicating because the replicator module and FTDLMare not forward processing. Snapshotting occurs by extracting data fromthe registrar's database and sending it to the entrant, which then addsit to the entrant's database. Before any snapshotting occurs, theentrant deletes all data in its database to ensure a clean copy.

Refinement 5b: If the entrant is using verify/update initialization, theentrant and registrar compare contents of their databases. Any corruptdata in the entrant's database is rectified; missing data is added(i.e., discrepancies are resolved). If the entrant contains new data,the entrant may create actions corresponding to this new data and markthose actions Red.

Refinement 5c: If the entrant is using backup/historical initialization,the registrar sends all Green actions after the last applied action atthe entrant to the entrant, which then applies those actions. Theregistrar does not send the contents of the ongoing queue.

Refinement 6: The registrar then sends Green and White actions andreplicator administrative data to the entrant. In the event that theentrant used backup/historical initialization, the historical data isalready present and only the replicator administrative data (includingGreen Lines, Red Cut, persistent quorum information) is sent. Uponreceipt of this replicator durable data, the entrant disconnects fromthe registrar. A flow control protocol is utilized so as not to exceedany network buffers or otherwise cause the network connection betweenthe registrar and the entrant to be dropped. The flow control is basedon the size of the actions. Furthermore, the registrar sends a modifiedversion of the registrar's Red Cut to the entrant, namely the GreenLines at the registrar converted to a Red Cut. This version is usedbecause the registrar does not send the contents of the action buffer;therefore the Red Cut on the entrant cannot know of those actions.

Refinement 7: The registrar rejoins the GCS group. The entrant contactsthe GCS system.

Refinement 8: The registrar and entrant recover.

Refinement 9: Once the entrant member has attained quorum, it'sreplicator durable data is up to date. The entrant member may thenbecome a registrar if so configured. Only Members are allowed to becomeregistrars if they have participated in at least one quorum; thisensures that their replicator durable data is at least as up to date asthe last known quorum. If the entrant created actions for data that onlyexisted on the entrant, those actions are marked green during recoveryin the usual manner.

Membership Leave Methodology

Membership Leave is the process by which a member leaves the persistentmembership and becomes a server. Typically, the application of aMembership Change message to the persistent membership results in amember being removed from the persistent membership and the removal ofadministrative data associated with the member. A number of refinementsto this membership leave methodology will now be described.

Voluntary leave: In a voluntary leave, a member sends a MembershipChange message which, when applied to the persistent membership, willremove the member from the persistent membership. In accordance with anembodiment of the present invention, prior to sending the MembershipChange message, the member's FTDLM must inhibit all in processtransactions by causing them to exception. Upon marking the MembershipChange message Green, the member can leave the GCS, which forces a viewchange causing the server to stop forward processing. The server is notallowed to modify the replicator durable data upon leave in the eventthat the server is used as the source for a Membership Join utilizingbackup/historical database initialization. Rather, initialization of thereplicator durable storage occurs upon initiating the membership joinprotocol. A member cannot leave the persistent membership if it isunable to send a Membership Change message, as occurs when the RCSM hasnot attained quorum. By requiring that the member be part of a quorum,it is ensured that at least one other member will record the exit of theleaving member, and therefore every member of the replication group willrecord the exit of the leaving member.

Eviction: Eviction occurs when a member is unable to voluntarily leavebecause it has crashed. In this case, another member must send aMembership Change message on behalf of the crashed member. A member mayneed to be evicted for several reasons: it crashed holding distributedlocks which are needed, the server has crashed unrecoverably (e.g.,hardware failure), or an entrant crashed unrecoverably during databaseinitialization. Once a member has been evicted from the persistentmembership, all locks held by that member are released; the local copyof the lock queues at every other member is updated to reflect this.

Recovery Methodology

Quorum: In one embodiment of the present invention, a dynamic linearvoting algorithm (e.g., such as the one described in “From Total Orderto Database Replication,” by Yair Amir and Ciprian Tutu, which wasincluded in the previously incorporated U.S. Provisional Application No.60/671,373; or the one described in “Dynamic voting algorithms formaintaining the consistency of a replicated database,” Jajodia andMutchler, 1990) with weighting in order to determine quorum. In one suchcase, every member has a fixed weight, chosen at the time the memberjoins the persistent membership. The weight is transmitted in theMembership Request message as well as in Membership Change messages andevery member's weight is stored in every member's persistent storage. Inone particular embodiment, quorum is attained if more than half of thesum of the weights of the members in the last quorum are present. Thequorum requirements can be loosened (thereby allowing more quorums) bynot counting members of the last quorum that have left or have beenevicted. Without excluding members that have left or have been evicted,overly weighted members that have left the persistent membership canprevent any further quorums. Likewise in a replication group where everymember has equal weight, it is impossible for the penultimate member toleave the persistent membership.

Determination of Most Up To Date Member: The most up to date method inthe replicator module may be unable to determine a most up to datemember if administrative data is removed upon change of the persistentmembership. The most up to date method in the replicator module assumesthat any Member that has no record of the last applied action from anarbitrary member must not have applied any actions from that arbitrarymember, and therefore must need all actions from that arbitrary member.This is an invalid assumption if the persistent membership changed whilea member was failed. In such a case, surviving members may have lessadministrative information than recovering members, leading to theinability to determine a most up to date member. To provide for thiscase, two versions of the persistent membership can be maintained: theactive and inactive persistent membership. The active persistentmembership contains all members that are currently part of thereplication group. The inactive persistent membership contains allmembers that have left the active persistent membership, but whoseremoval from the active persistent membership may not be known to allmembers yet. Members in the inactive membership are removed when theassociated Membership Change messages containing the inactive membershave been marked Green on all members; that is, when those messages canbe deleted (because they have been marked White). When members areremoved from the inactive membership, the associated replicatoradministrative data for those removed members is also removed.Therefore, that administrative data is no longer needed as all membershave applied the Membership Change message.

Recovery of Evicted Members: An evicted member does not know it has beenevicted until it recovers. Upon recovery, all members send both theactive persistent membership and the inactive persistent membership, andall store the latest versions of each. If, during recovery, thesurviving and recovering members determine an evicted member is present,all cease recovery until the next view change. The evicted member leavesthe GCS group, forcing a view change. A member considers itself evictedif it is not in either the active persistent membership or the inactivepersistent membership. Members that upon recovery determine that theyare in the inactive persistent membership recover up to the point thatthey apply their own leave Membership Change message. This allowsevicted members to be used as source for backup/historicalinitialization.

Initial Membership

The initial persistent membership may be agreed upon before recovery ofany member, or it may be empty. If empty, a server must be chosen to bethe initial member. In one particular embodiment of the presentinvention, this is accomplished via a Self Join method. The Self Joinmethod modifies the initial member's replicator durable storage to addthe initial member directly to the persistent membership. Uponmodification of the replicator durable storage the member contacts theGCS system, receives a view change, attains quorum, and enters forwardprocessing. The initial member can now act as a registrar for additionalmembers.

Alternatively, a set of servers may be designated as the initialmembers, as long as the durable storage at every server is appropriatelyinitialized the servers may recover together. The absence of previousquorum information invalidates the quorum algorithm. To provide aprevious quorum, each member during recovery simulates a previous quorumby considering the last quorum to be all the members in the initialpersistent membership. Therefore, the first quorum is the weightedmajority of the persistent membership.

The foregoing description of the embodiments of the invention has beenpresented for the purposes of illustration and description. It is notintended to be exhaustive or to limit the invention to the precise formdisclosed. Many modifications and variations are possible in light ofthis disclosure. It is intended that the scope of the invention belimited not by this detailed description, but rather by the claimsappended hereto.

1. A method for dynamically managing persistent membership of acoordination group in a distributed environment, the coordination groupincluding a number of sites included in a persistent membership, withone or more of the site including a database and a registrar, the methodcomprising: receiving, by a registrar of a first site that is a memberof the persistent membership, a Membership Request including a server IDand a name for an entrant seeking to become a member of the persistentmembership; sending, by the registrar of the first site, a MembershipChange message to other sites in the coordination group; receiving, bythe registrar of the first site, the Membership Change message in totalpersisted order; proceeding, by the registrar of the first site, withinitializing the entrant database in response to the persistentmembership being changed; modifying, by the registrar of the first site,the persistent membership for the entrant; responsive to the registrarreceiving the Membership Request, disconnecting the registrar of thefirst site from the coordination group and inhibiting, by a distributedlock manager, updates to a first database, while cotemporaneous to theinhibition of the updates (1) the registrar of the first site isinitializing an entrant database for the entrant and (2) other sites ofthe coordination group update at least one database in the coordinationgroup; and reconnecting, by the registrar of the first site, to thecoordination group after initialization of the entrant database.
 2. Themethod of claim 1 wherein should connection between the registrar of thefirst site and the entrant be severed, the entrant can reconnect to anyavailable registrar via a group communication system providing anextended virtual synchrony service to the coordination group, andrecover.
 3. The method of claim 1 wherein initializing the entrantdatabase is carried out using backup/historical initialization, and theMembership Request includes an action ID of a last action that wasapplied to the entrant database, the action ID uniquely identifying thepersistent membership.
 4. The method of claim 3 wherein in response tothe registrar of the first site not containing historical updateinformation for this action ID, the method further comprises refusing,by the registrar of the first site, to proceed.
 5. The method of claim 1further comprising: acquiring, by the registrar of the first site, adistributed lock on the coordination group and the name for the entrant;examining, by the registrar of the first site upon receiving thedistributed lock, the persistent membership to ensure that no other siteutilizes the name for the entrant; and refusing, by the registrar of thefirst site, to proceed with initializing the database in response to thename for the entrant being already in use.
 6. The method of claim 5wherein acquiring the distributed lock is carried out by a faulttolerant distributed lock manager associated with the registrar of thefirst site, the method further comprising: aborting, by the faulttolerant distributed lock manager, any transactions in process;releasing, by the fault tolerant distributed lock manager, any locksheld by the registrar of the first site; and disconnecting, by theregistrar of the first site, from a group communication system providingan extended virtual synchrony service to the coordination group therebyallowing the registrar of the first site to provide a consistentsnapshot to the entrant.
 7. The method of claim 1 wherein initializingthe entrant database is carried out using snapshot initialization, andthe entrant database is initialized with contents of the first databaseof the first site.
 8. The method of claim 1 wherein initializing theentrant database is carried out using verify/update initialization, theinitializing further comprising: comparing contents of the entrantdatabase and the first database of the first site; resolvingdiscrepancies; and in response to the entrant including new data,creating, by the entrant, actions corresponding to the new data andindicating total persisted order of those actions is not yet known. 9.The method of claim 1 wherein initializing the entrant database iscarried out using backup/historical initialization, the initializingfurther comprising: sending, by the registrar of the first site, allGreen actions after last applied action at the entrant to the entrant,wherein Green actions are those whose total persisted order is known andconfirmed; and applying, by the entrant, those sent Green actions. 10.The method of claim 1 further comprising: inhibiting, by the distributedlock manager, transactions during database initialization.
 11. Themethod of claim 1 wherein the registrar of the first site can onlyservice an entrant if the registrar of the first site has attainedquorum based on a quorum requirement, and the quorum requirement can beloosened to allow more quorums by not counting previous quorum membersthat have left or have been evicted.
 12. The method of claim 1 whereinthe registrars of the sites are allowed to service the MembershipRequest from the entrant only if the sites have participated in at leastone quorum.
 13. The method of claim 1 further comprising: using a flowcontrol protocol so as not to exceed any network data flow limits. 14.The method of claim 1 further comprising: connecting, by the entrant, tothe coordination group after initialization of the entrant database. 15.A machine-readable medium encoded with instructions, that when executedby one or more processors, cause the processor to carry out a processfor dynamically managing persistent membership of a coordination groupin a distributed environment, the coordination group including a numberof sites included in a persistent membership, with one or more of thesites including a database and a registrar, the process comprising:receiving, by a registrar of a first site that is a member of thepersistent membership, a Membership Request including a server ID and aname for an entrant seeking to become a member of the persistentmembership; sending, by the registrar of the first site, a MembershipChange message to other sites in the coordination group; receiving, bythe registrar of the first site, the Membership Change message in totalpersisted order; proceeding, by the registrar of the first site, withinitializing the entrant database in response to the persistentmembership being changed; modifying, by the registrar of the first site,the persistent membership for the entrant; responsive to the registrarreceiving the Membership Request, disconnecting the registrar of thefirst site from the coordination group and inhibiting, by a distributedlock manager, updates to a first database while cotemporaneous to theinhibition of the updates (1) the registrar of the first site isinitializing an entrant database for the entrant and (2) other sites ofthe coordination group are able to update at least one database in thecoordination group; and reconnecting, by the registrar of the firstsite, to the coordination group after initialization of the entrantdatabase.
 16. The machine-readable medium of claim 15, the processfurther comprising: inhibiting, by the distributed lock manager,transactions during database initialization.
 17. The machine-readablemedium of claim 15 wherein the registrar of the first site can onlyservice an entrant if the registrar of the first site has attainedquorum based on a quorum requirement, and the quorum requirement can beloosened to allow more quorums by not counting previous quorum membersthat have left or have been evicted.
 18. The machine-readable medium ofclaim 15 further comprising: connecting, by the entrant, to thecoordination group after initialization of the entrant database for theentrant.